<html>
<head><meta charset="utf-8"><title>aws codebuild for the release process · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html">aws codebuild for the release process</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="201215162"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215162" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215162">(Jun 17 2020 at 23:59)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> in an unrelated chat <span class="user-mention" data-user-id="198819">@David Barsky</span> brought up that aws codebuild (aws's own CI) is a thing, and I thought we might use it to move the release process out of RCS</p>



<a name="201215244"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215244" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215244">(Jun 18 2020 at 00:00)</a>:</h4>
<p>the main difference between running it on fargate and codebuild is that codebuild has 64gb of local disk space, while fargate only has 20gb (and our build process take more than that)</p>



<a name="201215318"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215318" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215318">(Jun 18 2020 at 00:01)</a>:</h4>
<p>I wanted to eventually move the release process to fargate by rewriting it to process just a couple of files at the time instead of batch-downloading all of them at the start, but that would've taken a bunch of development and testing time</p>



<a name="201215392"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215392" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215392">(Jun 18 2020 at 00:02)</a>:</h4>
<p>and moving the release process out of RCS would allow us to downsize that instance (finally!) and potentially grant more people access in the future (as we can manage it through IAM)</p>



<a name="201215404"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215404" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215404">(Jun 18 2020 at 00:02)</a>:</h4>
<p>RCS instance is pretty small I think?</p>



<a name="201215406"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215406" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215406">(Jun 18 2020 at 00:02)</a>:</h4>
<p>Just two core</p>



<a name="201215426"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215426" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215426">(Jun 18 2020 at 00:03)</a>:</h4>
<p>I remember it being a ton expensive</p>



<a name="201215428"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215428" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215428">(Jun 18 2020 at 00:03)</a>:</h4>
<p>But yeah, this seems plausible. FWIW for releases efs on fargate should work just fine which gives us unlimited storage</p>



<a name="201215442"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215442" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215442">(Jun 18 2020 at 00:03)</a>:</h4>
<p>It worked poorly for perf because sticking a database on efs is pretty painful, but for releases it should work well</p>



<a name="201215445"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215445" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215445">(Jun 18 2020 at 00:03)</a>:</h4>
<p>nope</p>



<a name="201215489"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215489" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215489">(Jun 18 2020 at 00:04)</a>:</h4>
<p>the problem is bandwidth</p>



<a name="201215502"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215502" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215502">(Jun 18 2020 at 00:04)</a>:</h4>
<p>Does fargate have poor bandwidth?</p>



<a name="201215509"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215509" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215509">(Jun 18 2020 at 00:04)</a>:</h4>
<p>EFS unless you store terabytes on it</p>



<a name="201215536"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215536" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215536">(Jun 18 2020 at 00:05)</a>:</h4>
<p>Hm I'm confused, you're saying that efs is super slow?</p>



<a name="201215538"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215538" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215538">(Jun 18 2020 at 00:05)</a>:</h4>
<p>other than bursts it's 50kb/s per GB stored</p>



<a name="201215548"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215548" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215548">(Jun 18 2020 at 00:05)</a>:</h4>
<p>unless we pay a ton for reserved bandwidth</p>



<a name="201215561"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215561" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215561">(Jun 18 2020 at 00:05)</a>:</h4>
<p><a href="https://docs.aws.amazon.com/efs/latest/ug/performance.html#bursting">https://docs.aws.amazon.com/efs/latest/ug/performance.html#bursting</a></p>



<a name="201215640"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215640" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215640">(Jun 18 2020 at 00:06)</a>:</h4>
<p>We might fit into burst bucketing</p>



<a name="201215648"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215648" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215648">(Jun 18 2020 at 00:06)</a>:</h4>
<p><a href="/user_uploads/4715/6k71et56o3eRQ5Jiam-RecUC/2020-06-18-02-06-37.png">2020-06-18-02-06-37.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/6k71et56o3eRQ5Jiam-RecUC/2020-06-18-02-06-37.png" title="2020-06-18-02-06-37.png"><img src="/user_uploads/4715/6k71et56o3eRQ5Jiam-RecUC/2020-06-18-02-06-37.png"></a></div>



<a name="201215652"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215652" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215652">(Jun 18 2020 at 00:06)</a>:</h4>
<p>Anyway, I'm not really opposed</p>



<a name="201215667"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215667" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215667">(Jun 18 2020 at 00:07)</a>:</h4>
<p>granted, I still need to fully evaluate it</p>



<a name="201215673"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215673" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215673">(Jun 18 2020 at 00:07)</a>:</h4>
<p>but it seems promising</p>



<a name="201215688"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215688" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215688">(Jun 18 2020 at 00:07)</a>:</h4>
<p>FWIW getting experience with aws ci seems good regardless</p>



<a name="201215696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/aws%20codebuild%20for%20the%20release%20process/near/201215696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/aws.20codebuild.20for.20the.20release.20process.html#201215696">(Jun 18 2020 at 00:07)</a>:</h4>
<p>Since maybe that's the future for e.g. sponsored builders or so</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>